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1.  Summary  of  Achievements 


During  the  41-month  period  June  1,  1986  -  November  30,  1989,  the  following 
was  achieved  under  support  from  Grant  AFOSR-86-0172: 

(i)  Seventeen  Computer  Sciences  Department  technical  reports  were  written  and 
submitted  for  outside  publication.  Five  have  already  appeared  in  print,  three 
have  been  accepted  for  publication  and  nine  are  under  review. 

(ii)  Four  Ph.D.  dissertations  were  supervised.  Two  dissertations  are  complete,  the 
third  will  be  completed  in  June  1990  and  the  fourth  in  December  1991. 

(iii)  Forty-one  technical  talks  were  presented  by  the  Principal  Investigator  outside 
the  University  of  Wisconsin  at  scientific  meetings  and  other  universities  on 
research  work  supported  by  this  grant. 
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2.  Summary  of  Technical  Results 


Seventeen  reports  were  written  in  the  41-month  period  beginning  June  1,  1986, 
under  support  from  Grant  AFOSR-86-0172.  Titles  and  abstracts  of  these  reports  are 
given  in  Section  3  of  this  report. 

The  principal  area  of  research  undertaken  under  this  gram  is  the  parallel  solution 
of  very  large  sparse  linear  programs.  The  ground  work  for  this  approach  was  laid  in 
[1]  where  the  inherently  serial  successive  overrelaxation  (SOR)  scheme  for  solving 
very  large  sparse  linear  programs  was  parallelized.  With  the  serial  version  of  the  SOR 
scheme  [5]  we  were  able  to  solve  linear  programs  with  as  many  as  125,000  variables 
and  500,000  constraints  on  a  MicroVax  II  in  less  than  72  hours.  We  are  not  aware  of 
any  general  linear  programming  packages  that  can  solve  linear  programs  of  this  size. 
The  parallel  approach  proposed  in  [1]  required  a  relaxation  factor  to  to  be  restricted  to 
2 

the  interval  0  <  to  — where  o  is  a  positive  number  depending  on  the  coupling 

among  the  parallel  blocks  of  the  problem.  If  the  problem  is  loosely  coupled,  o 
approaches  zero.  In  [2,  5]  a  more  efficient  parallel  SOR  scheme  was  proposed  in 
which  to  is  restricted  to  the  classical  interval  0  <  to  <  2.  This  resulted  in  larger  step- 
sizes  and  a  faster  algorithm.  Speedup  efficiency  of  the  algorithm  as  tested  on  the 
CRYSTAL  token-ring  multicomputer  and  the  Sequent  Balance  21,000  multiprocessor 
was  in  the  65  to  100  percent  range.  Another  interesting  and  useful  result  was  that  a 
serial  implementation  of  our  parallel  SOR  scheme  was  more  efficient  than  the  original 
serial  SOR  scheme. 

In  [3]  a  new  and  practically  useful  characterization  of  the  solution  set  of  a  con¬ 
vex  program  was  obtained.  In  particular  we  have  shown  that  the  gradient  of  the 
objective  function  of  a  convex  program  is  constant  on  the  solution  set  of  the  problem. 
Furthermore,  the  solution  set  lies  in  an  affine  subspace  orthogonal  to  this  constant  gra¬ 
dient,  and  is  in  fact  in  the  intersection  of  this  affine  subspace  with  the  feasible  region. 
This  result  can  be  used  to  give  simple  and  computationally  useful  characterizations  of 
solution  sets  of  a  number  of  important  problems. 

In  [4]  methods  are  proposed  for  the  solution  of  the  non-monotone  (hence  non- 
convex  and  very  difficult)  complementarity  problem  A  guaranteed  lower  bound  to  a 
least  2-narm  solution  of  this  problem  is  given. 

In  [6]  a  parallel  asynchronous  SOR  is  proposed  for  the  solution  of  symmetric 
linear  complementarity  problems  and  linear  programs.  In  [7]  a  two-stage  (SOR) 
method  for  solving  the  symmetric  linear  complementarity  problem  is  given  that  can 
solve  large  problems  that  cannot  be  tackled  by  existing  algorithms.  In  [8]  a  parallel 
implementation  of  Lemke’s  algorithm  is  given  for  solving  the  linear  complementarity 
problem  with  speedup  efficiency  of  essentially  100%. 

In  [9]  new  asynchronous  algorithms  are  proposed  for  the  solution  of  the  sym¬ 
metric  linear  complementarity  problems  tested  on  the  Sequent  Symmetry  S81  mul¬ 
tiprocessor  with  speedup  efficiency  in  the  43%-91%  range.  The  test  results  clearly 
show  the  superiority  of  the  asynchronous  algorithms  over  their  synchronous  counter¬ 
parts. 
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In  [10]  error  bounds  are  given  which  guarantee  how  for  a  given  arbitrary  point  is 
from  the  solution  set  of  a  nondegenerate  monotone  linear  complementarity  problem. 
Nondegeneracy  induces  a  simplicity  in  the  bounds  that  cannot  be  obtained  otherwise. 
These  bounds  are  important  theoretically  and  useful  computationally. 

In  [11]  we  establish  convergence  of  an  asynchronous  parallel  algorithm  for  solv¬ 
ing  the  nonsymmetric  linear  complementarity  problem.  Numerical  results  show  that 
asynchronism  improves  computational  efficiency. 

In  [12]  we  show  that  under  appropriate  conditions,  perturbing  a  convex  program¬ 
ming  objective  function  is  equivalent  to  minimizing  the  perturbation  function  on  the 
solution  set  of  the  original  unperturbed  convex  program.  This  result  has  useful  com¬ 
putational  implications. 

In  [13]  we  use  linear  programming  to  diagnose  breast  cancer  data.  A  computer 
program  based  on  our  work  is  currently  in  use  at  University  of  Wisconsin  Hospitals, 
and  has  successfully  diagnosed  98%  of  new  cases  encountered. 

In  [14]  we  compare  our  linear  programming  approach  of  [13]  with  a  neural  let- 
work  approach  on  breast  cancer  data.  The  results  indicate  superiority  of  the  unear 
programming  approach. 

In  [15]  we  establish  a  practical  results  which  shows  that  under  a  nondegeneracy 
assumption,  linearizing  the  objective  function  around  a  point  sufficiently  close  to  a 
solution,  will  yield  an  exact  solution  in  one  step.  This  leads  to  finite  termination  of 
any  convergent  algorithm  if  this  linearization  is  performed  periodically. 

In  [16]  we  show  that  complexity  of  the  pattern  recognition  problem  varies  from 
being  NP-complete  to  being  in  P,  depending  on  the  norm  employed.  This  result  has 
real  practical  computational  significance. 

In  [17]  we  establish  convergence  and  test  an  interior  proximal-point  algorithm 
for  linear  programming.  On  a  standard  set  of  test  problems  it  outperforms  MINOS  5.0 
by  a  factor  of  5. 


3.  Papers  Written  under  Support  of  Grant  AFOSR-86-0172 
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Reproduced  below  are  abstracts  of  the  above  reports  given  in  the  same  order  as 
the  above  Usttng: 

1 .  A  parallel  successive  overrelaxation  (SOR)  method  is  proposed  for  the  so¬ 

lution  of  the  fundamental  symmetric  linear  complementarity  problem.  Convergence  is 
established  under  a  relaxation  factor  which  approaches  the  classical  value  of  2  for  a  loosely 
coupled  problem.  The  parallel  SOR  approach  is  then  applied  to  solve  the  symmetric  linear 
complementarity  problem  associated  with  the  least  norm  solution  of  a  linear  program. 


2 .  Gradient  projection  saeceashre  overrelaxation  (GP-SOR)  algorithm  is  pro¬ 
posed  for  the  solution  of  symmetric  linear  complementarity  problems  and  linear  programs. 
A  key  distinguishing  feature  of  this  algorithm  is  that  when  appropriately  parallelized,  the 
relaxation  factor  interval  (0,  2)  is  not  reduced.  In  a  previously  proposed  parallel  SOR 
schemes  the  substantially  reduced  relaxation  interval  mandated  by  the  coupling  terms  of 
the  problem  often  led  to  slow  convergence.  The  proposed  parallel  algorithms  are  applied 
to  finding  the  least  2-norm  solution  of  linear  programs.  Efficiency  of  the  algorithm  is  in 
the  50  to  100  percent  range  as  demonstrated  by  computational  results  on  the  CRYSTAL 
token-ring  multicomputer  and  the  Sequent  Balance  21000  multiprocessor. 

3.  By  means  of  elementary  arguments  we  first  show  that  the  gradient  of  the 
objective  function  of  a  convex  program  is  constant  on  the  solution  set  of  the  problem. 
Furthermore  the  solution  set  lies  in  an  affine  subspace  orthogonal  to  this  constant  gradi¬ 
ent,  and  is  in  fact  in  the  intersection  of  this  affine  subspace  with  the  feasible  region.  As 
a  consequence  we  give  a  simple  polyhedral  characterization  of  the  solution  set  of  a  convex 
quadratic  program  and  that  of  a  monotone  linear  complementarity  problem.  For  these  two 
problems  we  can  also  characterize  a  priori  the  boundedness  of  their  solution  sets  without 
knowing  any  solution  point.  Finally  we  give  an  extension  to  nonsmooth  convex  optimiza¬ 
tion  by  showing  that  the  intersection  of  the  subdifierentials  of  the  objective  function  on 
the  solution  set  is  nonempty  and  equals  the  subdiiferential  of  the  objective  function  at 
any  point  in  the  relative  interior  of  the  optimal  solution  set.  In  addition,  the  solution  set 
lies  in  the  intersection  with  the  feasible  region  of  an  affine  subspace  orthogonal  to  some 
subgradient  of  the  objective  function  at  a  relative  interior  point  of  the  optimal  solution 
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4.  We  show  that  a  least  2-norm  solution  of  a  general  linear  complementarity 

problem  (LCP)  can  be  obtained  by  solving  a  sequence  of  perturbed  quadratic  programs. 
The  norm  of  the  solutions  of  the  perturbed  problems  approaches  monotonically  from  below 
the  norm  of  a  least  2-norm  solution  of  the  LCP,  as  the  perturbation  parameter  approaches 
zero.  For  sufficiently  large  value  of  the  perturbation,  the  quadratic  program  is  strongly 
convex  and  easily  solved  by  Lemke’s  method.  This  yields  a  guaranteed  lower  bound  to  the 
norm  of  a  least  2-norm  solution.  For  some  LCP’s,  even  non-monotone  ones,  the  perturbed 
quadratic  program  may  give  a  solution  to  the  LCP  for  a  finite  value  of  the  perturbation 
parameter.  For  monotone  LCP’s  the  perturbed  quadratic  program  yields  the  least  2-norm 
solution  for  a  finite  value  of  the  perturbation  parameter  if  and  only  if  the  least  2-norm 
solution  of  the  linearized  LCP  equals  that  of  the  original  LCP. 

5.  Serial  and  parallel  successive  overrelaxation  (SOR)  methods  are  proposed  for 

the  solution  of  the  augmented  Lagrangian  formulation  of  the  dual  of  a  linear  program.  With 
the  proposed  serial  version  of  the  method  we  have  solved  linear  programs  with  as  many  as 
125  000  constraints  and  500,000  variables  in  less  that  72  hours  on  a  Micro Vax  II.  A  parallel 
implementation  of  the  method  was  carried  out  on  a  Sequent  Balance  21000  multiprocessor 
with  speedup  efficiency  of  over  65%  for  problem  sizes  of  up  to  10,000  constraints,  40,000 
variables  and  1,400,000  nonzero  matrix  elements. 


6.  We  preKnt  a  parallel  asynchronous  successive  overrelaxation  algorithm 

for  the  solution  of  symmetric  linear  complementarity  problems  and  linear  programs. 
A  distinguishing  feature  of  this  algorithm  is  that  processors  need  not  communicate 
after  each  update  of  the  solution  vector  and  therefore  processor  idle  time  can  be 
avoided.  The  proposed  parallel  algorithm  is  applied  to  finding  least  2 -norm  solu¬ 
tions  of  linear  programs.  Improvement  is  observed  over  the  synchronized  version  of 
the  algorithm,  the  parallel  gradient  projection  successive  overrelaxation  algorithm. 


7.  We  propose  a  two-stage  successive  overrelaxation  (SOR)  algorithm 

for  solving  the  symmetric  linear  complementarity  problem.  After  the  first  SOR 
preprocessing  stage  this  algorithm  concentrates  on  updating  a  certain  prescribed 
subset  of  variables  which  is  determined  by  exploiting  the  complementarity  property. 
We  demonstrate  that  this  algorithm  successfully  solves  problems  with  as  many  as 
10,000  variables  which  cannot  be  tackled  by  other  current  algorithms. 


8 .  We  propose  a  parallel  implementation  of  the  classical  Lemke's  algorithm 

for  solving  the  linear  complementarity  problem.  The  algorithm  is  designed  for 
a  loosely  coupled  network  of  computers  which  is  characterized  by  relatively  high 
communication  costs.  We  provide  an  accurate  prediction  of  speedup  based  on  a 
simple  operation  count.  The  algorithm  produces  speedups  near  p,  where  p  is  the 
number  of  processors,  when  tested  on  large  problems  as  demonstrated  by  computa¬ 
tional  results  on  the  CRYSTAL  token-ring  multicomputer  and  the  Sequent  Balance 
21000  multiprocessor. 


9 .  Convergence  is  established  for  asynchronous  parallel  successive  overrelaxation 

(SOR)  algorithms  for  the  symmetric  linear  complementarity  problem.  For  the  case  of  a 
strictly  diagonally  dominant  matrix  convergence  is  achieved  for  a  relaxation  factor  interval 
of  (0,2)  with  line  search,  and  (0,  lj  without  line  search.  Computational  tests  on  the 
Sequent  Symmetry  S81  multiprocessor  give  speedup  efficiency  in  the  43%-91%  range  for 
the  cases  for  which  convergence  is  established.  The  tests  also  show  superiority  of  the 
asynchronous  SOR  algorithms  over  their  synchronous  counterparts. 
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1 0 .  Error  bounds  and  upper  Lipschitz  continuity  results  are  given  for  mono¬ 

tone  linear  complementarity  problems  with  a  nondegenerate  solution.  The  existence 
of  a  nondegenerate  solution  considerably  simplifies  the  error  bounds  compared  with 
problems  for  which  all  solutions  are  degenerate.  Thus  when  a  point  satisfies  the  linear 
inequalities  of  a  nondegenerate  complementarity  problem,  the  residual  that  bounds 
the  distance  from  a  solution  point  consists  of  the  complementarity  condition  alone, 
whereas  for  degenerate  problems  this  residual  cannot  bound  the  distance  to  a  solution 
without  adding  the  square  root  of  the  complementarity  condition  to  it.  This  and  other 
simplified  results  are  a  consequence  of  the  polyhedral  characterization  of  the  solution 
set  as  the  intersection  of  the  feasible  region  {z\Mz  +  q  >0,  z  >  0}  with  a  single  linear 
affine  inequality  constraint. 


1 1 .  Convergence  is  established  for  the  multi-sweep  asynchronous  parallel 

successive  overrelaxation  (SOR)  algorithm  for  the  nonsymmetric  linear  complemen- 
-•.arity  problem.  The  algorithm  was  originally  introduced  in  [4]  for  the  symmetric 
linear  complementarity  problem.  Computational  tests  show  the  superiority  of  the 
multi-sweep  asynchronous  SOR  algorithm  over  its  single-sweep  counterpart  on  both 
symmetric  and  nonsymmetric  linear  complementarity  problems. 


1 2 .  This  paper  concerns  a  characterization  of  the  finite  perturbation  property  of  a  convex  pro¬ 

gram.  When  this  property  holds,  finite  perturbation  of  the  objective  function  of  a  convex  program  leads  to 
a  solution  of  the  original  problem  which  minimizes  the  perturbation  function  over  the  set  of  solutions  of  the 
original  problem.  This  generalizes  a  finite  termination  property  of  the  proximal  point  algorithm  for  linear 
programs  and  characterizes  finite  Tikhonov  regularization  of  convex  programs. 
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13.  Multisurface  pattern  separation  Is  a  mathematical  method  for  dis¬ 
tinguishing  ^between  elements  of  two  pattern  sets.  Bach  element  of  the  pat¬ 
tern  set  Is  comprised  of  various  scalar  observations.  In  this  paper,  we  use 
the  diagnosis  of  breast  cytology  to  demonstrate  the  applicability  of  this 
method  to  medical  diagnosis  and  decision  making.  Eleven  cytological  charac¬ 
teristics  of  breast  fine  needle  aspirates  (fna)  reported  to  differ  between 
benign  and  malignant  samples  were  graded  (1  to  10)  at  the  time  of  sample  col¬ 
lection.  Nine  characteristics  were  found  to  differ  significantly  between 
benign  and  malignant.  Mathematically,  these  values  for  each  sample  were 
represented  by  a  point  In  a  nine-dimensional  space  of  real  variables.  Benign 
points  were  separated  from  malignant  ones  by  planes  determined  by  linear  pro¬ 
gramming.  Correct  separation  was  accomplished  In  369  of  370  samples  (201  be¬ 
nign  and  169  malignant).  In  the  one  misclaasifled  malignant  case,  the  fna 
cytology  was  so  definitely  benign  and  the  cytology  of  the  excised  cancer  so 
definitely  malignant  that  we  believe  the  tumor  was  missed  on  aspiration. 
This  mathematical  method  is  applicable  to  other  medical  diagnostic  and 
decision-making  problems. 

1 4 .  TWo  computer-driven  expert  systems  were  trained  to  correctly  diag¬ 
nose  369  fine-needle  breast  aspirates  (fna's)  on  the  basis  of  nine  cytological 
descriptive  parameters  and  were  tested  cn  24  newly  obtained  fna's  (eighteen 
benign  end  six  malignant).  One  system,  generated  by  multisurface  pattern 
separation,  correctly  classified  all  training  and  test  samples,  while  the 
other,  generated  by  a  connect lonlst  algorithm  (neural  network) ,  Incorrectly 
classified  one  of  the  benign  test  samples  as  malignant  (l.e.  one  false 
positive) .  Even  though  the  oultlsurface  expert  system  appears  better  able  to 
diagnose  new  data  patterns  than  the  neural  network  system,  both  expert  systems 
aid  breaat  cytological  diagnosis  and  can  serve  as  models  far  other  applica¬ 


tions 
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1 5.  Wt  characterize  the  property  of  ob. mining  a  solution  to  a  convex  program  by  minimizing  over 

the  feasible  region  a  linearization  of  the  objective  function  at  any  of  its  solution  points  (Minimum  Principle 
Sufficiency).  For  the  case  of  a  monotone  line**  complementarity  problem  this  MPS  property  is  completely 
equivalent  to  the  existence  of  a  nondegenerate  solution  to  the  problem.  For  the  case  of  a  convex  quadratic 
program  the  MPS  property  is  equivalent  to  the  span  of  the  Hessian  of  the  objective  function  being  contained 
in  the  normal  cone  to  the  feasible  region  at  any  solution  point  plus  the  cone  generated  by  the  gradient  of  the 
objective  function  at  any  solution  point.  This  in  turn  is  equivalent  to  the  quadratic  program  having  a  weak 
sharp  minimum.  An  important  application  of  the  MPS  property  is  that  minimizing  on  the  feasible  region 
a  linearization  of  the  objective  function  at  a  point  in  a  neigborhood  of  a  solution  point  gives  an  exact 
solution  of  the  convex  program.  This  leads  to  finite  termination  of  convergent  algorithms  that  periodically 
minimize  such  a  linearization. 


16.  A  dvririnn  problem  — i oristnH  with  a  fundamental  paaowwi  modal  far  linearly 

inseparable  pattern  eete  is  shown  to  b«  NP-complete.  ftnnthnrnnnrnmiTliiiiilillhil  mqilnjssn  i  in 
norm  inrtiid  of  tha  2-norm,  can  bo  solved  in  polynomial  time  by  solving  2n  linear  programs,  where 
n  is  the  (usually  small)  dimension  slity  of  tbs  pattern  space.  An  effective  LP-b  seed  finite  algorithm  is 
pcopessd  far  solving  tbs  Utter  model.  The  algorithm  is  employed  to  obtain  a  nooconvex  piecewise- 
linear  function  far  separating  points  representing  measurements  made  on  fine  needle  aspirates  taken 
Grom  benign  and  malignant  human  breasts.  A  computer  program  trained  on  388  samples  has  correctly 
diagnosed  each  of  48  new  samples  encountered  and  is  currently  in  use  at  the  University  of  Wisconsin 
Hospitals. 


1 7.  An  interior  point  algorithm  for  obtaining  a  proximal  point  solution 
of  a  linear  program  is  presented.  Results  from  our  implementation  of 
this  algorithm  have  been  very  encouraging.  For  36  test  problems  in¬ 
cluding  32  NETLIB  problems,  we  obtain  a  total  time  speedup  of  5.6 
over  the  MINOS  5.0  simplex  package.  We  also  describe  an  imple¬ 
mentation  of  our  algorithm  for  linear  programs  with  upper-bounded 
variables,  such  as  the  multicommodity  Patient  Distribution  System 
models  of  the  Military  Airlift  Command.  We  have  been  able  to  solve 
some  of  these  multicommodity  problems  with  8-figure  accuracy  and 
speednp  of  as  much  as  24  over  the  MINOS  5.0.  Furthermore  our  run 
times  on  the  Astronautics  ZS-1  are  comparable  with  those  of  AT&T’s 
KORBX  times  for  some  of  the  problems. 


-12- 


4.  Ph.D.  Dissertations  Supervised 


Student 

Dissertation  Title 

Completion 

Date 

Michael  C.  Ferris 

Weak  sharp  minima 
and  penalty  functions 
mathematical  programming 

August  1989 

Lois  Brady 

Condition  constants 
for  solutions  of 
convex  inequalities 

August  1988 

Rudy  Setiono 

Interior  penalty 
methods  for  linear 
and  quadratic  programs 

June  1990 

Kristin  Bennett 

Pattern  recognition  via 
mathematical  programming 

December  1991 

